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Executive Summary 


Purpose 


American taxpayers invested more than $60 billion in federal funds in 
military and civilian research and development (R&D) efforts in 1996. The 
private sector invested more than $110 billion that same year. The 
technological advancements resulting from these efforts are a critical 
factor in improving the productivity of American workers and, 
correspondingly, the nation’s standard of living. However, while the 
contribution of R&D to technological advancement is widely recognized, 
there is no widely accepted method of measuring the results of that 
research. 


To facilitate discussions of the adequacy of the funding and of the results 
of the R&D, the Subcommittee on Technology, House Committee on 
Science, asked GAO to evaluate the various indicators that are used to 
measure the results of R&D. Specifically, this report discusses the strengths 
and limitations of the input and output indicators used by the federal and 
private sectors to measure the results of R&D. This report also provides a 
historical perspective on spending for research. 


Background 


The commitment to reduce the federal deficit is forcing the Congress to 
reexamine the value of programs across the federal government. Although 
scientific research is often considered to be intrinsically valuable to 
society, there is pressure on all federal agencies, including science 
agencies, to demonstrate that they are making effective use of the 
taxpayers’ dollars. This greater emphasis on results is evident in the 
passage of the Government Performance and Results Act of 1993 (GPRA). 
The act fundamentally seeks to shift the focus of federal management and 
accountability from a preoccupation with staffing, activity levels, and 
tasks completed to a focus on results—that is, the real difference that 
federal programs make in people’s lives. 


The experts in research measurement have tried for years to develop 
indicators that would provide a measure of the results of R&D. However, 
the very nature of the innovative process makes measuring the 
performance of science-related projects difficult. For example, a wide 
range of factors determine if and when a particular R&D project will result 
in commercial or other benefits. It can also take many years for a research 
project to achieve results. 


C6 2 a 
Results in Brief 


The amount of money spent on research and development, the primary 
indicator of the investment in research, is useful as a measure of how 
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much research is being performed. Having been refined over many years, 
these data are generally available for the research efforts in both the 
public and private sectors. However, the level of spending is not a reliable 
indicator of the level of results achieved by research. 


Unlike the situation with the input measures of research and development, 
there is no primary indicator of the outputs. Output indicators include 
quantitative analyses of return on investment, patents granted, and other 
outputs as well as qualitative assessments based on peer review. The 
companies that GAO spoke with collect data on various output indicators 
but, in general, make limited use of them in their investment decisions. 
Instead, the companies emphasized that research and development 
contribute directly to their “bottom line.” Because companies are 
profit-oriented, many of the indicators tracked by the private sector 
cannot be directly applied to the federal government. Experiences from 
pilot efforts made under the Government Performance and Results Act 
have reinforced the finding that output measures are highly specific to the 
management and mission of each federal agency and that no single 
indicator exists to measure the results of research. 


Principal Findings 


Funding Indicates 
Research Activity but Does 
Not Measure the Results of 
Research 


Funding has been used as the primary input indicator for decades. 
Whether a policymaker is interested in basic research, applied research, or 
development, the amount of money spent in that area is taken as an 
indication of how much research is being performed. The major 
advantages of using expenditure data as an indicator are that they are 
easily understandable, readily available, and have been, in general, 
consistently gathered over time. In addition, spending on different projects 
in different research areas can be measured according to the same unit, 
dollars, making comparisons between projects straightforward. 


The amount of funding, however, does not provide a good indication of the 
results of research. Companies told GAo that they are focusing more of 
their spending on short-term R&D projects than on long-term projects. 
However, the impacts of that change in focus are unclear. The reduced 
funding levels may not reflect the fact that the R&D efforts are being 
performed with greater efficiency. For example, one way in which the 
federal government and the private sector have tried to use R&D resources 
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more efficiently and effectively is through consortia with universities or 
other companies. By combining their research activities, the companies 
attempt to avoid expensive duplication and learn from each other. 


R&D Output Indicators 
Can Provide Limited 
Information About the 
Results of R&D 


Because of the difficulties in identifying the impacts of research, 
decisionmakers have developed quantitative and qualitative indicators as 
proxies to assess the results of R&D activity. The strengths and limitations 
are evident in both types of indicators. The current quantitative indicators 
focus mainly on return on investment, patenting rates, and 
bibliometrics—the study of published data. While implying a degree of 
precision, these indicators were not originally intended to measure the 
long-term results of R&D. Qualitative assessment provides detailed 
information, but it relies on the judgments of experts and may be 
expensive. 


Because of these difficulties, the companies interviewed by GAO stressed 
marketplace results rather than R&D output indicators. While varying in the 
types of indicators they collect, they emphasized the difficulties in 
measuring R&D’s specific contribution to a company’s overall performance. 
For example, one company stated that because so many people have been 
involved in a product’s evolution, it is difficult to separate the contribution 
of the research unit from that of other units. All of the companies 
interviewed have increased their expectation that R&D contribute directly 
to their profitability, but instead of increasing their efforts at measuring 
R&D results, they have shifted the responsibility for R&D decisions to the 
business units. However, many of the R&D output measures tracked by the 
private sector do not apply directly to the federal government. In 
particular, while facing the same increasing cost pressures as the private 
sector, the federal government cannot rely on the profit motive to guide its 
decisions. 


The GPRA requires the executive agencies to develop their annual R&D plans 
with suitable performance measures. The Research Roundtable, a group of 
federal researchers and managers representing a cross-section of R&D 
departments and agencies, warned about the difficulties of quantifying the 
results of R&D and the potential for incorrect application with subsequent 
harm to scientific endeavors. The Army Research Laboratory, which was 
designated a pilot project for performance measurement under the act, has 
developed a multifaceted approach using quantitative indicators, peer 
review, and customer feedback to evaluate the results of R&D. 
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This report contains no recommendations. 


Recommendations 


Because this report focuses broadly on the R&D of both the federal and 
private sectors, and not on the effort of individual agencies, GAo did not 
submit a draft of this report to federal agencies for their review and 
comment. 


Agency Comments 


Page 5 GAO/RCED-97-91 Measuring Performance 


Contents 


Executive Summary - 

Chapter 1 : P 

Intromuen The Process of Innovation and the Use of R&D Indicators 8 

ntroauction Past Efforts to Evaluate R&D Impacts 10 

The Government Performance and Results Act 11 
Objectives, Scope, and Methodology 12 

Chapter e R&D Spending Gi Indicati f I tive Effort il 

: pending Gives an cation of Innovative Effo 

R&D Spending Data R&D Spending Is Not a Good Indicator of R&D Results 15 

Provide Some Spending Patterns Show a Greater Emphasis on Short-Term 7 

Information About nggaarch 

Innovative Activity 

but Not About R&D 

Results 

Chapter 3 C t Indicators Have Strengths and Limitati fs 

urrent Indicators Have Strengths and Limitations 

R&D Output The Private Sector’s Emphasis on Marketplace Results Limits 22 

Indicators Can Lessons for the Federal Government 

Provide Limited Federal Science Agencies Are Still Exploring Ways to Measure 26 

; the Impacts of R&D 

Information About 

R&D Results 

App endixes Appendix I: Historical Perspective on Research Spending 28 
Appendix II: Major Contributors to This Report 34 

Figure Figure I.1: U.S. R&D Spending, 1953-96 30 


Page 6 GAO/RCED-97-91 Measuring Performance 


Contents 


Abbreviations 

ARPA Advanced Research Projects Agency 

DOD Department of Defense 

GAO General Accounting Office 

GPRA Government Performance and Results Act 

IRI Industrial Research Institute 

NIH National Institutes of Health 

NSF National Science Foundation 

NASA National Aeronautics and Space Administration 
OECD Office of Economic Cooperation and Development 
OTA Office of Technology Assessment 

R&D research and development 


Page 7 


GAO/RCED-97-91 Measuring Performance 


Chapter 1 _ 


Introduction — 


The Process of 
Innovation and the 
Use of R&D Indicators 


Over $180 billion was spent on research and development (R&D) in the 


United States in 1996.' Most of that amount was spent by industry and the 
federal government—$113 billion and $62 billion, respectively; the balance 
was spent by universities and other nonprofit organizations. The leading 
experts in the study of research indicators agree that R&D has a significant, 
positive effect on economic growth and the overall standard of living. 
However, because of the complexity of linking the results of R&D to its 
economic impacts, there is no widely accepted method of measuring the 
results of R&D spending in either industry or the federal government. The 
commitment to reduce the federal budget deficit is forcing the Congress 
and the executive branch to undertake a basic reexamination of the value 
of programs across the federal government. It is also placing pressure on 
all federal agencies, including the civilian science agencies, to clearly 
demonstrate that they are making effective use of the taxpayers’ dollars. 
The Government Performance and Results Act of 1993 (GPRA) provides a 
legislative vehicle for the agencies to use as they seek to demonstrate and 
improve their effectiveness. Equally important, if successfully 
implemented, GPRA should help the Congress make the difficult funding, 
policy, and program decisions that the current budget environment 
demands. 


Researchers have developed their own terminology for describing the 
process of transforming R&D into economic results. In its simplest form, 
the theory underlying both public and private decision-making has been 
that innovative activity positively affects economic performance. 
Innovation can be thought of as the development and application of a new 
product, process, or service. It can include the use of an existing product 
in anew application or the development of a new device for an existing 
application. Innovation encompasses many activities, including scientific, 
technical, and market research; product, process, or service development; 
and manufacturing and marketing to the extent that they support the 
dissemination and application of the invention. Innovation is a 
combination of invention and commercialization. Invention describes the 
initial conception of a new product, process, or service, but not the act of 
putting it to use. Commercialization refers to the attempt to profit from 
innovation through the sale or use of new products, processes, and 
services. 


'These figures are based on preliminary 1996 statistics reported by R&D performers to the National 
Science Foundation. 
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This description as well as the traditional views of innovation has been 
strongly influenced by the linear model of innovation, which says that 
innovation proceeds sequentially through the stages of basic research, 
applied research, development, manufacturing, and marketing.” This 
model assumes that basic research serves as the source of innovation and 
that new scientific knowledge initiates a chain of events culminating in the 
development and sale of a new product, process, or service. In this model, 
basic research is the most uncertain part of the process; once basic 
research is conducted, innovation and commercialization can proceed. 
The model suggests that the firms with the best technology will likely be 
the first to market and win the lion’s share of profits. 


The simplicity of this model makes it particularly useful in policy 
discussions. Other models may be more accurate, but they provide a more 
complex explanation of the relationship between science and the 
commercialization of new technology. These models, such as the 
“chain-linked model,” include feedback loops that allow for interaction 
among the different stages of the linear model. These models also reflect 
the fact that the ideas for new inventions or changes to existing products 
often arise from the recognition of new market opportunities, advances in 
manufacturing capabilities, or advances in technology independent of 
progress in the underlying science. 


Real-world examples show that technological breakthroughs can precede 
as well as follow basic research. In many cases, science is not the source 
of innovation. The Wright brothers, for example, developed the first 
airplane without an understanding of aerodynamic theory, and Chester 
Carlson developed the first xerographic copier without a thorough 
understanding of photoconductive materials. These inventions have 
resulted in considerable research into aerodynamic theory and materials 
science, respectively, as scientists and engineers attempted to improve the 
original invention. 


In studying complex processes or concepts such as innovation, it is not 
always possible to measure them directly. As a result, researchers turn to 
the use of “indicators.” Indicators point to or illustrate the process or 


The National Science Foundation uses the following definitions in its resource surveys: Basic research 
has as its objective to gain a more comprehensive knowledge or understanding of the subject under 
study, without specific applications in mind. Applied research is aimed at gaining knowledge or 
understanding to determine the means by which a specific, recognized need may be met. Development 
is the systematic use of the knowledge or understanding gained from research directed toward the 
production of useful materials, devices, systems, or methods. The Foundation recognizes the 
limitations of this classification scheme but continues to use these categories to maintain historical 
consistency, among other reasons. 
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concept in question but do not directly measure it. For example, in order 
to determine an object’s temperature, one can use a thermometer to 
measure it directly. However, when trying to measure something as 
complex as the health of a country’s economy, one relies on different 
indicators, such as unemployment rates, stock market averages, or trade 
balances that do not provide a direct measurement of economic health but 
do give an indication of its status. In the case of innovation, various “input” 
and “output” indicators are used that are based on the linear model of 
innovation. The following chapters of this report are broken down 
according to these two sets of indicators. Chapter 2 is concerned with R&D 
spending, or expenditure data, which is the most widely used input 
indicator of innovation, and chapter 3 focuses on some widely used output 
indicators. 


Past Efforts to 
Evaluate R&D 
Impacts 


For almost two decades, numerous reports have documented the 
difficulties of quantifying the results of R&D. As noted above, the 
identification of the economic and social effects of research involves 
complex issues of measurement, analysis, and interpretation. The 
following efforts may help to give perspective to the present concerns 
about measuring R&D results. 


In 1979, we reported on a wide range of factors that make the 
measurement of R&D results difficult.? The report noted that R&D 
expenditures are undertaken for a variety of reasons. Some attempt to 
develop new knowledge; others are directed at meeting needs, such as 
national defense, for which there is no commercial market; and still others 
are directed at lowering the cost of products. Furthermore, some projects 
produce revolutions; others produce nothing. Most important, R&D is only 
one input into a complex process. Thus, we concluded that there is no 
possibility of eliminating the role that judgment plays in the allocation of 
federal R&D resources. 

In 1986, the Office of Technology Assessment (OTA) issued a detailed 
report that questioned the utility of the effort to quantify R&D returns.’ 
According to oTa, the fundamental stumbling block to placing an economic 
value on federal R&D is that improving productivity or producing an 
economic return is not the primary justification for most federal R&D 
programs. The report added that the attempts to measure the economic 
return to federal R&D are flawed because many of the research outputs, 


3 Assessing the “Output” of Federal Commercially Directed R&D (GAO/PAD-79-69, Aug. 1979). 


“Research Funding as an Investment: Can We Measure the Returns? Office of Technology Assessment, 
72 pp. (Apr. 1986). 
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The Government 
Performance and 
Results Act 


such as national defense, cannot be assigned an economic value. The 
report also noted that in industry, where one might expect quantitative 
techniques to prevail because of the existence of a well-defined economic 
objective, oTA found a reliance on subjective judgment and good 
communications between R&D, management, and marketing staffs. 

In 1996, responding to concerns about the implementation of GPRA, the 
National Science and Technology Council issued a report that stressed the 
limited role of quantification in measuring the results of RaD.° It stated that 
the insufficiency of quantitative measures per se is one reason why other 
sources of evidence, such as merit review of past performance, narrative 
discussion, and descriptions of outstanding accomplishments and more 
typical levels of achievement, should be included in annual performance 
reports. The report concluded that the cornerstone of world-class science 
will continue to be merit review with peer evaluation, while the science 
community works to develop the measurement tools and other methods 
needed to assess the contributions of fundamental science. 

At the international level, the Organization of Economic Cooperation and 
Development (OECD) is also grappling with these questions. Beginning with 
the first edition of the Standard Practice for Surveys of R&D (the “Frascati 
Manual”) in the 1960s, orcp has been developing international frameworks 
for the measurement of R&D inputs. In connection with its more recent 
effort to develop new output indicators (the “Oslo Manual”), OECD stated 
that the new indicators and the underlying statistics usually take two or 
more decades to reach general acceptance and regular collection and 
publication. The participants at a 1996 oEcD conference extensively 
discussed the organization’s efforts to improve the quality of innovation 
indicators. At this stage, these indicators appear to be most useful in 
helping researchers study and describe the process of innovation. 


In response to questions about the value and effectiveness of federal 
programs, GPRA seeks to shift federal agencies’ focus away from such 
traditional concerns as staffing, activity levels, and tasks completed 
toward a focus on program outcomes—that is, the real difference the 
federal program makes in people’s lives. In the GPRA context, an “outcome 
measure” assesses the results of a program activity compared to its 
intended purpose. An “output measure,” according to GPRA, tabulates, 
calculates, or records the level of activity or effort and can be expressed in 
a quantitative or qualitative manner. The output indicators discussed in 
our report could be considered either outcome measures or output 
measures, depending on the context. For example, return-on-investment 


> Assessing Fundamental Science, National Science and Technology Council (July 1996). 


Page 11 GAO/RCED-97-91 Measuring Performance 


Chapter 1 
Introduction 


Objectives, Scope, 
and Methodology 


calculations could be outcome measures in a business context since 
gaining the maximum return on investment is the intended purpose of a 
business. Patenting rates would be output measures in some businesses 
because patents could serve as one measure of the level of activity of a 
research unit. 


The act recognizes how difficult it is to state the goals and measure the 
results of some programs. While the law encourages the use of objective 
measures of performance, it authorizes agencies—with the approval of the 
Office of Management and Budget—to use alternative, subjective 
measures of performance. Also, instead of having GPRA take effect 
immediately after its passage in 1993, the Congress allowed for a period of 
time for the government to learn how to implement the act. As part of the 
learning process, the act called for the use of pilot projects in performance 
measurement. Among the approximately 70 agencies or parts of agencies 
that participated in pilot projects, one addresses scientific research. Upon 
the full implementation of the act, the executive branch agencies are 
required to devise plans that are outcome-oriented. The act calls for the 
agencies to develop three documents: a strategic plan, an annual 
performance plan, and an annual performance report. 


In response to the Subcommittee on Technology’s request, our objective 
was to review various indicators that are used to measure the results of 
R&D. Specifically, this report discusses the relative strengths and 
limitations of the input and output indicators used by the federal and 
private sectors to measure the results of R&D as well as the claim that 
industry focuses on short-term profitability rather than long-term R&D 
needs. This report also provides a historical perspective on research 
spending. (See app. I.) 


The impacts of innovation are widely studied in the public and private 
sectors as well as in academia. Our work relied on a limited number of 
experts in each of these sectors to provide us with the prevailing 
understanding of and latest developments in the area. As a result, our 
review does not provide an exhaustive examination of R&D measures nor 
does it answer the question of how R&D should be measured. It does, 
however, as agreed with your offices, consist of information on the 
strengths and limitations in the use of these indicators as well as anecdotal 
information based on interviews with leading R&D companies. We also 
reviewed the relevant literature. 
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We interviewed a number of experts, including a former IBM Vice President 
for Research and National Science Foundation (NSF) Director, a former 
Executive Director of the Manufacturing Forum, a former Associate 
Director of the White House Office of Science and Technology Policy, the 
Director of the Special Projects Office for the Army Research Laboratory, 
and the former Chief Financial Officer for Apple Computer. We also 
conducted six teleconferences with company officials who were typically 
at the director level. We interviewed representatives of General Electric, 
Lucent Technologies (formerly Bell Labs), Dow Chemical, 
Eastman-Kodak, IBM, and Microsoft Corporation. These companies spent 
from $5 billion to $800 million on R&D in 1995. We chose companies having 
among the largest total R&D budgets, in terms of dollars spent, because we 
believed that their experiences would be the most relevant to the needs of 
the federal government. 


Our work also covered the available data on R&D spending and output 
indicators provided by the leading data-gathering organizations in this 
field—the OECD, NSF, and the Industrial Research Institute (irr). Our 
collaboration with the OECD included participation in an OECD conference 
on science and technology indicators. Throughout the course of this work, 
we also interviewed the NSF staff responsible for publishing the Science 
and Engineering Indicators. 


We performed our audit work from June 1996 through February 1997 in 
accordance with generally accepted government auditing standards. 
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R&D Spending Data Provide Some 
Information About Innovative Activity but 
Not About R&D Results 


R&D spending data provide an indication of how much research is being 
performed but do not provide a measure of the impacts of that spending. 
Spending data have been used for budgeting purposes; however, they have 
also been used as an indicator of the level of innovative activity within a 
nation or company. The use of R&D spending data as an indicator has a 
number of advantages. For example, it reduces the innovation process to a 
single figure for the purposes of discussion. In addition, the data-gathering 
methods have been refined over many years and are generally reliable over 
time. However, the level of spending is not a reliable indicator of the level 
of research results. For example, companies told us that they are focusing 
more of their spending on short-term R&D projects than on long-term 
projects, but the impacts of that change in emphasis are unclear. The use 
of spending data is more appropriate for discussions of R&D spending 
priorities than of the effectiveness and impacts of R&D spending levels. 


R&D Spending Gives 
an Indication of 
Innovative Effort 


Traditionally, R&D expenditures have been taken to indicate the “amount” 
of innovative activity that is occurring within a country or a firm. One of 
the advantages of using expenditure data in this way is that it simplifies 
the discussion of the complex process of innovation to a single unit of 
measurement. Another advantage is that the use of dollars as the unit of 
measurement enables direct comparisons to be made. In addition, the 
gathering of spending data has been refined over many years, increasing 
the data’s reliability and relevance to policy-making. 


This straightforward rationale—the more R&D spending, the more 
innovative activity—is the primary advantage to using expenditure data in 
policy discussions. Its simplicity and close ties to the linear model of 
innovation allow it to be readily understood by those with little specialized 
knowledge, making it appealing in policy discussions. These same 
simplifying characteristics may have led to its use in other areas. In some 
contexts, countries and companies are categorized according to their 
technological sophistication on the basis of their R&D spending levels; little 
attention is given to other factors. 


Another advantage arises from the common use of “dollars” in the R&D 
spending data. This usage enables the spending in different research areas 
to be compared according to the same units. These straightforward 
comparisons are useful in demonstrating the priorities of the nation at 
large. For example, recent U.S. R&D expenditure data show that a 
reduction in defense-related R&D was Somewhat counterbalanced by an 
increase in federal support for civilian R&D programs, including those 
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R&D Spending Is Not 
a Good Indicator of 
R&D Results 


aimed at improving the diagnosis and treatment of disease, cleaning up the 
environment, and enhancing technological competitiveness and economic 
prosperity. In addition, converting different foreign currencies to dollars 
allows for international comparisons of research. After allowing for the 
variation of inflation over time, funding data depict historical patterns of 
real expenditures. 


The accuracy and policy relevance of spending data are also important 
advantages to their use as indicators. The typical source of these data is 
the widely cited Science and Engineering Indicators document published 
periodically by the National Science Board. The Science Resources 
Studies Division of NsF has been gathering data to use in Indicators for 
many years, and many improvements have been made in the accuracy of 
the data.® For example, NsF has sponsored the Survey of Industrial Rap 
since 1953 as a source of data for Indicators. Recent improvements to this 
survey include selecting samples annually (rather than less frequently) and 
increasing the sample size from approximately 14,000 to nearly 24,000 
firms. NSF took these steps to account more accurately for the 
establishment of R&D-performing entities in the survey universe and to 
survey more fully and accurately the R&D performed by nonmanufacturing 
firms. In addition, NsF is constantly searching for ways to make Indicators 
more responsive to policymakers’ needs. NSF began to make adjustments 
in its surveys when it recognized that there was a need to supply more 
information on the service sector to policymakers because the survey 
historically had focused on the manufacturing industries in which R&D 
performance had been heavily concentrated in the past. 


The use of spending data is limited in its relevance to the impacts of R&D. 
There is some correlation between the level of R&D spending and 
innovative success. For example, if fewer research projects are performed, 
then companies and countries forgo the potential benefits of the research. 
However, spending alone does not guarantee innovative success because 
many additional factors figure into the innovation process and have 
important effects on the resulting outputs. The reality of the process of 
innovation is much more complex than expenditure data alone can reveal. 


The usefulness of R&D spending data as an indicator of results is limited 
because the data measure the amount of resources a firm or a nation 


®Some problems with the data may still exist, however. For example, NSF itself notes that the data on 
trends in the private sector’s basic research contain anomalous spending spikes in 1986 and 1991. The 
inconsistencies in the data appear to derive from changes made in the NSF survey in those years. For 
example, in 1991 the survey was expanded to cover a broader array of nonmanufacturing firms. 
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dedicates to innovation, but not its ability to convert that effort into 
successful products, processes, and services. Accounts in the press have 
noted that companies are proud of high R&D spending levels because 
well-focused R&D generally pays off in long-term revenue growth and 
profits. However, more is not always better, as shown by the companies 
that ranked among the leaders for R&D spending and disappeared shortly 
thereafter. For example, two companies called Xonics and Transitron 
Electronic ranked as the top companies in R&D spending per employee for 
1984 and 1986, respectively; in the same years that they achieved their top 
ranking, one company was forced to file for Chapter 11 protection and the 
other company dissolved. 


There is great uncertainty in research and development investments. The 
processes leading to commercially viable and socially useful technologies 
are complex and involve substantial non-R&D factors. One company official 
told us that it was impossible to determine which business function was 
most important to the success of a new product—research, marketing, or 
sales. While there is some correlation between the level of R&D spending 
and innovative success, spending alone does not guarantee success. One 
official from a high-technology company told us that his company is 
constantly evaluating research projects with respect to their targeted 
markets and estimating the expected return on investment. However, in 
one case the company lost approximately $100 million when the market 
would not support a newly developed product at the price the company 
expected. 


The usefulness of R&D spending indicators is also limited because the way 
in which innovative activities are structured and managed can be as 
significant as the amounts of resources devoted to them in determining 
their outcomes and effects on performance. Those nations or firms with 
extremely efficient innovation systems can outperform those that use 
greater R&D resources inefficiently. As a former director of NSF pointed out 
to us, streamlining research efforts could reduce both bureaucratic and 
financial overhead costs and make up for spending reductions. 


In addition, falling spending levels may hide the greater efficiencies that 
might be realized by leveraging research. One company official told us that 
although his company was cutting back on long-term research, it was 
relying more on its relationships with federal and university laboratories 
for similar work. Collaboration between firms—through joint ventures, 
consortia, and contracting—has recently been on the rise as firms attempt 
to efficiently distribute risk, pool their resources, and tap into external 
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Emphasis on 
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expertise. The magnitude of alliance formation is difficult to gauge, as are 
the implications for the innovation and commercialization of new 
technologies in the United States. However, one academic told us that if 
the rhetoric about the efficiencies coming from cooperative R&D is true, 
then no one should be too disappointed with recent drops in R&D spending. 


Federal and private R&D spending patterns reflect the changing conditions 
in the country and the world at large. The share of basic research 
supported by private sources peaked in the early post-World War II period, 
driven by thriving domestic and international markets. However, recent 
surveys and anecdotal evidence suggest that increased international 
competitive pressures have forced companies to emphasize short-term 
development over long-term research. Company officials echoed this shift 
and suggested that the focus on short-term research was part of a push 
toward more relevancy in their R&D departments. They pointed out that 
their business units are determining their research needs to a greater 
extent today than they have in the past. 


Some companies told us that this strategy could be viewed as 
short-sighted. Others defended this strategy. For example, one company 
official told us that his company emphasizes short-term research because 
“competition is short-term.” He stated that in today’s competitive 
environment, once a market is lost, it is gone forever. 
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Current Indicators 
Have Strengths and 
Limitations 


Quantitative and qualitative indicators have been developed to evaluate 
R&D activities and their results, but both types of indicators have strengths 
and limitations. Our interviews with a number of companies showed that 
the private sector stresses marketplace results rather than relying on 
output indicators. Because of the companies’ profit orientation, many of 
the indicators tracked by the private sector cannot be directly applied to 
the federal government. In response to the GPRA, the federal science 
agencies are exploring new ways to quantify the impacts of research. 
However, it is too early to tell whether new performance measures can be 
developed and whether they will meet the needs of the Congress. 


Because of the difficulties in identifying the impacts of research, 
decisionmakers in the public and private sectors typically have chosen to 
measure outcomes using a variety of proxies. These quantitative and 
qualitative indicators have strengths and limitations. To illustrate these 
strengths and limitations, we looked at three of the most frequently cited 
quantitative indicators: return on investment, patents issued, and 
bibliometrics. While these indicators imply a degree of precision, they 
were generally not designed to measure the long-term results of R&D 
programs, nor are they easily adaptable to such a purpose. Qualitative 
assessment provides detailed, descriptive information, but it depends on 
subjective judgments and may be costly. 


Return on Investment 


This indicator aims at measuring the sales and profits resulting from 
investments in R&D; as such, it addresses one of the fundamental concerns 
about the value of such investments. However, a variety of factors, such as 
the complexity of the innovation process and its inherently long time 
frames, pose serious obstacles to the calculation of these returns. The 
literature dealing with return on investment is replete with words of 
caution against quantifying R&D results. NsF’s Science and Engineering 
Indicators 1996 pointed out that not only is much of this information 
unobtainable or ambiguous, but many of the gains from research are 
simply monetarily intangible. 


Experts on the R&D process have stated frequently that the long time 
periods and multiple inputs involved make the task of calculating the 
return on basic research especially difficult. Productivity growth may lag 
20 years behind the first appearance of research in the scientific 
community, and the lag for interindustry effects may be 30 years. A more 
serious impediment, however, is the fact that outcomes are often not 
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Patents 


directly traceable to specific inputs or may result from a combination of 
such inputs. The National Aeronautics and Space Administration (NASA) 
and the National Bureau of Standards (now the National Institute for 
Standards and Technology) attempted to measure the economic impacts 
and benefits of certain of their technologies in the 1970s and early 1980s. 
The studies at the Bureau were discontinued, according to staff, because 
of serious theoretical and methodological problems. As with the Bureau’s 
studies, NASA’s studies evoked serious criticisms and were likewise 
discontinued. 


Despite the difficulties in calculating return on investment, the leading 
researchers in this field agree that R&D offers high private and social 
returns in terms of high productivity. One recent survey of 63 studies 
found that R&D activity achieves, on average, a 20- to 30-percent annual 
return on private (industrial) investments.’ 


Patents show certain strengths as useful indicators in measuring technical 
change and inventive input and output over time. For example, they can 
reveal a variety of trends involving ownership and levels of activity in 
technical areas. According to NSF, the data concerning ownership show 
that the federal share of patents averaged 3.5 percent of the total number 
of U.S. patents during 1963 through 1980 but declined thereafter. 


In addition, the data concerning a country’s distribution of patents by 
technical area have proved to be a reliable indicator of a nation’s 
technological strengths as well as its direction in product development. 
For example, the three most emphasized U.S. patent categories for 
inventors show specific contrasts between U.S. and foreign patents. U.S. 
inventors obtained most of their patents in wells, mineral oils, and certain 
areas of surgery. Japanese inventors focused their efforts on certain areas 
of communications, organic compounds, and refrigeration. Patent activity 
can be used to pinpoint potentially important changes. In 1980 through 
1987, U.S. inventors led all other foreign inventors in radio and television 
patents, but in 1987 the United States lost its front position to Japanese 
inventors in this area. 


Despite their usefulness as indicators of broad national and international 
trends in various industries and areas of research, patents also have 
several intrinsic drawbacks. Inconsistency across industries in the number 
of patents granted is a major limitation that results from the wide 


"Nadiri, M.I. Innovations and Technological Spillovers (1993). 
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variations in the propensity to patent inventions. Consequently, according 


to NSF, it is not advisable to compare patenting rates between different 
technologies or industries. Inconsistency in quality is a second drawback. 
The aggregated patent statistics do not distinguish between those that led 
to major innovations and those that led to minor improvements. 
Incompleteness is a further limitation. Many inventions are not patented at 
all, and trade secrecy may become a preferred alternative to patenting. 
Another limitation is that patents do not lend themselves to the evaluation 
of the most significant results achieved by an R&D program. They can 
provide intermediate measures of progress, but they are not usually the 
purpose for which the research was undertaken. 


In addition, the use of patents as a measure of federal R&D effectiveness 
may be hampered by their limited relevance. The 1996 report entitled 
Assessing Fundamental Science by the National Science and Technology 
Council noted that any use of patent counts should be undertaken only 
with a full awareness of their limitations. A recent academic study of 
patents and the evaluation of R&D also commented on the extraordinarily 
limited applicability of patent evaluation to government-performed R&D. 
Both studies pointed to the relatively low level of federal patenting 
activity. The academic study noted that most government laboratories are 
granted only one or two patents per year, and only a few of them patent 
extensively. It concluded that for government laboratories, one may 
question the overall wisdom of evaluating public R&D with private 
techniques.® 


Bibliometrics 


A third area of effort in developing quantitative measurements involves 
bibliometrics, or the study of published data. Bibliometrics counts 
citations in an attempt to address questions of productivity, influence, and 
the transfer of knowledge. Its most appropriate use is in quantifying the 
work of researchers whose results are published. Thus, it may be 
especially applicable in areas such as basic research where the results are 
more often published than protected by firms. However, its usefulness as a 
measure of research results remains somewhat controversial. 


We believe that the use of bibliometrics as a source of information on the 
quality of the publications or the citations being counted needs to be 


5In its October 11, 1996 issue, Science magazine noted the French government’s announcement that 
patent records will form a part of the evaluation of publicly funded researchers. The new proposal, 
according to the magazine, is controversial because it might upset the proper balance between basic 
and applied research by favoring those who work in fields that are immediately applicable at the 
expense of people doing basic research. 
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approached with caution. Although bibliometric indicators can be 
weighted by publication or other quality measure, the frequency of 
citation, for example, provides no indication of the level of research 
innovation. Another limitation is the problems that arise in 
interdisciplinary comparisons of results. Some critics have gone so far as 
to say that bibliometric findings should not be used in science policy work 
until the problems with the analysis of citations are addressed. 


In addition, the relevance of bibliometric analysis to decision-making by 
the federal government appears very limited. One expert noted that a 
recent comprehensive review of bibliometrics shows the sparsity of 
bibliometric studies for evaluations of the impact of research reported by 
the federal government. Another pointed out that few federal agencies use 
bibliometric analysis as an evaluative tool. One of the few is the National 
Institutes of Health (NIH), which uses this method to evaluate the 
effectiveness of its different institutes, the comparative productivity of 
NIH-sponsored research and similar international programs, and the 
responsiveness of NIH’s research programs to their congressional mandate. 


Peer Review 


Recognizing the limitations of quantitative indicators, the National Science 
and Technology Council concluded that it makes sense to track relevant 
measures but that they cannot supplant the essential element of expert 
judgment. Peer review, the most important form of qualitative assessment, 
uses technical experts to judge R&D results on the basis of the expert’s 
evaluation of the quality of research. However, peer review has serious 
shortcomings; it generally depends on criteria that are inherently difficult 
to measure and on subjective judgment that is vulnerable to bias. 


Peer review has been used extensively in the selection of proposed 
research projects. To a lesser extent, it has also been used to evaluate R&D 
impacts. Peer review has come to be viewed by some observers as the best 
assurance that quality criteria will prevail over social, economic, and 
political considerations, while others view it as an element of elitism in 
science that tends to discount such concerns as economic significance. 


Its major strength is its ability to bring together the leading experts in the 
area of concern. Most peer review procedures require a minimum of three 
reviewers; if the review involves a more ambitious scope of coverage 
(such as an entire agency), dozens of reviewers may be involved. The 
process of selecting the peer reviewers varies. One of the chief 
responsibilities of the professional staff in science agencies such as NSF 
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and NIH is to stay in touch with a specialized community of scientists who 


are qualified to judge the agency’s activities. However, others rely on 
in-house managers who are not active researchers. 


The major limitations of peer review are twofold. First, the perception of 
quality depends largely on the expertise of the panel members. It is based 
on the judgment of experts about a proposal or a set of research-related 
results. Generally, a final judgment will depend on the collective weight of 
the different opinions. Frequently, a numerical rating scale—such as 1 for 
poor through 5 for excellent—is used. Despite the appearance of precision 
conferred by a specific number, the numbers represent the best of 
sometimes widely differing judgments. Consequently, although peer 
review has been a mainstay in judging science for over three centuries, 
questions remain about ways of improving it. For instance, according to 
one academic, to improve “validity and reliability,” research needs to be 
done on the optimal numbers of reviewers and on the advisability of 
training people to perform peer reviews. 


Second, peer review evaluation of completed or ongoing R&D projects is a 
more thorough and expensive process than peer review for the purpose of 
selecting proposals for funding. According to one study, the cost of a 
2-day, 10-person, face-to-face NSF merit review panel is in the 
neighborhood of $20,000. Another study concluded that if this method 
were applied annually to all federal research programs, the cost in 
reviewer time alone would be enormous. For example, the Army Research 
Laboratory has contracted for a peer review of its activities; the contract 
calls for a 3-year review directed by the National Research Council at 
approximately $650,000 per year. 


The Private Sector’s 
Emphasis on 
Marketplace Results 
Limits Lessons for the 
Federal Government 


The private-sector companies we interviewed varied in terms of the types 
of quantitative and qualitative R&D indicators that they collect, but in 
general they made limited use of these indicators in their decisions. Many 
companies stressed the difficulties involved in measuring the contribution 
of R&D to the firm’s overall performance using return on investment, 
patents issued, and other R&D output measures. All of the firms mentioned 
that they were increasing R&D that contributes directly to the bottom line 
of the firm. Thus, they shifted the responsibility for R&D decisions to the 
business-unit level so that the R&D would be tied more directly to the 
profits of those units. The private sector’s experience offers general 
lessons to the federal government in terms of ensuring that the R&D 
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contributes directly to the mission of the organization, although the 
specific output measures do not apply directly to federal R&D efforts. 


Companies Cited 
Difficulties in Measuring 
the Results of R&D 


Companies told us that measuring the return on R&D investment is very 
difficult. Companies stated that one factor making measurement more 
difficult is the long time lag between the research and any revenue that 
might be earned. Companies also stated that because so many people have 
been involved in a product’s evolution, it is difficult to separate the 
contribution of the research unit from that of other units. 


Companies also mentioned difficulties with some of the other indicators 
that they track. For example, one indicator was alternatively labeled a 
“vitality index” or “product innovation metric,” which reflected the share 
of the firm’s products that could be considered new products. This 
measure provided an indication of how rapidly the company was 
incorporating new ideas and research into its products. Several company 
officials mentioned that this measure had to be applied carefully because 
of the problem of defining a “new” product; some products are completely 
different from their predecessors, while others might incorporate cosmetic 
changes. 


Firms Increased Their 
Attention to R&D’s 
Contribution to the Bottom 
Line 


One issue that was mentioned by all of the firms in our discussions was 
the increasing emphasis on the relevance of R&D to business needs and the 
“bottom line.” These comments came up in a number of contexts. For 
example, some firms mentioned that they look for research with a 
shorter-term payoff in order to have a greater impact on getting new 
products into the marketplace. Other firms cited the increased emphasis 
on applied R&D, or the small amount of research that they perform that 
could be called basic research. A common element was that these firms 
were attempting to reduce what might be called benevolent research: 
projects that benefit the industry or the nation but do not have a payoff to 
the firm. 


However, a number of the firms noted that this emphasis on the bottom 
line does not necessarily create the opportunities for breakthrough 
products. All of the firms either reserved a certain fraction of their 
research funding for these types of projects or developed processes that 
made it possible to continue some level of research in areas that might not 
be directly aligned with any particular product line. In this context, one 
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firm mentioned the importance of cooperation with the federal 


government and universities in the pursuit of fundamental research. 


Firms Have Reduced 
Emphasis on R&D Output 
Measures 


While this increased emphasis on getting the most out of R&D might be 
expected to lead to greater efforts to measure the results of research, most 
of the firms that we spoke to responded by changing the organization so 
that measuring R&D outputs was no longer so important. By shifting the 
responsibility for research decisions to the business units in the firm that 
make use of the research outputs, the companies have sidestepped the 
need for centralized indicators of the quality of research. If the business 
units believe that a particular R&D project would increase their profits, the 
firm would budget for that R&D. If the business units do not perform up to 
expectations in terms of their profitability, the entire unit would be 
responsible. 


For example, one firm shifted from a policy of centrally directed research 
to a policy in which the individual business units make the decisions on 
the appropriate research projects. Under the previous arrangement, the 
various business units were assessed to support the central R&D efforts; 
under the new arrangement, the units pay only for those projects that they 
think are valuable. This shift was designed to make the research more 
responsive to the needs of the business units, in that these units do not pay 
for the research unless they find it useful. This shift also greatly reduced 
the emphasis on developing R&D output indicators in the central laboratory 
because each of the business units would be reviewed on the basis of its 
profitability. The reasoning is that if these units were making poor 
decisions on R&D projects, the unit’s overall profitability would decrease. 


Private Sector's Measures 
Do Not Directly Apply to 
the Federal Government 


Our interviews with private-sector firms suggest that many of the R&D 
output measures tracked by the private sector do not apply directly to the 
federal government. Many of these measures are directly related to the 
contribution of the R&D to the bottom line profitability of the firm. 
However, federal agencies do not operate in order to make a profit, but to 
accomplish a variety of other missions and goals. For example, agencies 
conduct R&D to support a variety of missions, such as maintaining national 
security or improving citizens’ health. R&D in these areas can contribute 
greatly to the quality of life in the United States, even if it has a negative 
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return on investment.’ Given these very different missions, there is also no 
reason to believe that any single measure is appropriate for different 
public-sector agencies. 


Despite the lack of specific measures that can be translated from the 
private to the public sector, there are general lessons to be learned from 
the private sector’s experience. Possibly the most important is the 
recognition that as the pressures on costs increased at many of the firms 
that we interviewed, the firms made significant efforts to ensure that R&D 
contributed directly to the bottom line. The federal government likewise 
faces increasing pressure on costs such as R&D expenditures but, unlike 
the private sector, cannot rely on the marketplace to ensure that the R&D 
contributes to the agency’s goals. Federal R&D is undertaken to support a 
variety of agency missions, and producing an economic return is not the 
primary justification for most federal R&D programs. In fact, the purpose of 
federal R&D is to promote research that is socially beneficial but may not 
be profitable for the private sector to pursue. Without this competitive 
marketplace to ensure the relevance of R&D, the federal agencies will 
continue to be challenged to develop better measures of the outputs of 
their R&D. 


The literature confirms this general finding of our discussions. For 
example, in 1995 the National Research Council reported on the results of 
a workshop on what the federal government might learn from the 
corporate experience in research restructuring and assessment.!° The 
Council invited senior corporate research managers from IBM, AT&T, Ford, 
and Xerox to discuss their experiences in this area. The report concluded 
that developing useful metrics and using them appropriately is a difficult 
problem, but it is not impossible. In addition, the participants were not 
successful in translating their private-sector experience into specific 
lessons about what can be measured and what makes sense to measure for 
the federal government. 


°The return-on-investment measure that is appropriate for the federal government would compare the 
total “social benefits” to the nation with the costs of the initial investment. Computing this measure 
has many of the difficulties of computing the return on investment for private firms, with the added 
complications associated with placing a value on national security, improved quality of life, and other 
intangible qualities. 


10Research Restructuring and Assessment: Can We Apply the Corporate Experience to Government 
Agencies?, 72 pp., National Research Council (1995). 
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Our July 1996 testimony on the implementation of GPRA noted that the 
Congress recognized that successful implementation will not come quickly 
or easily for many agencies." To help address the challenges of 
“measuring” the results of R&D programs, the Research Roundtable, a 
group of federal researchers and managers representing a cross-section of 
departments and agencies, has met periodically to share ideas and 
approaches for implementing GPRA. The Army Research Laboratory has 
also begun to address this issue in a pilot project for performance 
measurement under GPRA. 


The Research Roundtable has considered the extent to which R&D agencies 
can and should adopt a common approach to measuring performance. In 
1995, it issued a paper based on 6 months of discussions on the 
development of performance measures for research programs. Although 
the Roundtable stated that the results of a research program’s 
performance can be measured, it cautioned that at the same time, it is 
important to recognize the complexity of the cause-effect relationship 
between R&D and its results. It added that this complexity makes it difficult 
to establish quantifiable measures that consistently measure program 
performance. It also noted that such measures create a potential for 
incorrect application, which could lead subsequently to a detrimental 
effect on scientific endeavors. It warned that quantitative empirical 
demonstrations of such cause-effect relationships should not be required 
and are often not even possible. 


The Army Research Laboratory was designated a pilot project for 
performance measurement under GPRA. Of the more than 70 such pilot 
projects governmentwide, the laboratory was the only pilot project that 
addresses scientific research. As such, it attempted to break new ground 
in both the planning and the evaluation of basic and applied research. The 
Chief of the Army Research Laboratory’s Special Projects Office, who is 
mainly responsible for designing the laboratory’s approach to 
implementing GPRA, submitted a case study to the Office of Management 
and Budget in 1996." The case study outlines an approach that makes use 
of “three pillars:” metrics, peer review, and customer feedback. 


In the case study, the laboratory identified about 60 metrics, most of which 
measure input using fiscal, facilities, and personnel data. Some of the 


Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies 
(GAO/T-GGD/RCED-96-214, July 10, 1996). 


Brown, Edward A. “Applying the Principles of the Government Performance and Results Act to the 
Research and Development Function” (1996). 
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metrics, such as tasks completed, patents awarded, and articles published, 
measure output; none measure outcome. The laboratory views the 
measures as useful tools for understanding the functional health of the 
organization and the management of the laboratory but cautions that the 
information will not enable it to determine the real quality and impact of 
its R&D. The laboratory is relying more heavily on peer review of its 
research and on customer surveys for information about quality. 


According to the case study, the laboratory has contracted with the 
National Research Council of the National Academies of Science and 
Engineering to conduct a retrospective peer review of research over a 2- 
or 3-year period. The Council is to assemble a Technical Assessment 
Board that consists of six panels with 8 to 10 people, each of whom is of 
high repute within the technical community. These panels will appraise 
the quality of the laboratory’s technical and scientific efforts and, to a 
limited extent, productivity. 


The case study also described the use of customer feedback. The 
laboratory has identified a number of internal and external customers. For 
those customers to whom it delivers specific items, it uses a series of 
feedback questionnaires to determine their degree of satisfaction on a 1 to 
5 scale in terms of the quality, timeliness, and utility of the deliverable. For 
those customers who do not receive a specific, identifiable product, the 
laboratory is developing a Stakeholders’ Advisory Board of senior 
leadership and user representatives to provide first-hand guidance and 
feedback. 
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Since World War II, U.S. public and private research and development 
(R&D) spending patterns have reflected changing priorities as well as 
reactions to the changing national and international economies. Some of 
the more prominent events that have shaped spending in both sectors have 
been the Cold War and the recent international competitive pressures. 
Declines in spending in both sectors have been less frequent than 
increases. 


In his July 1945 report, Science—The Endless Frontier, Vannevar Bush, 
who headed the U.S. r&b effort during World War II, provided the rationale 
for federal support of both basic research and research related to national 
security, industry, and human health and welfare. His plan contributed to 
the legislation adopted in 1950 that established the National Science 
Foundation (NSF). By that time, however, the National Institutes of Health 
(NIH) already had control over most health-related research; the Office of 
Naval Research had taken on a major role in supporting academic 
research in the physical sciences; and the new Atomic Energy Commission 
had been assigned control of the R&D on nuclear weapons and nuclear 
power. NSF’s mission focused on supporting fundamental research and 
related educational activities. Its annual budget was less than $10 million 
until the late 1950s. In contrast, NIH’s annual budget, which had been less 
than $3 million at the end of the war, grew to more than $50 million by 
1950. 


The scope of federal R&D support grew in the decade after World War IL. 
Anxiety over the Cold War and the loss in 1949 of the U.S. monopoly in 
nuclear weapons led to expanded R&D programs in the Army and in the 
newly established Air Force and to a continuing buildup in support for 
nuclear weapons R&D in the Atomic Energy Commission. On the civilian 
side, R&D programs were established or expanded in fields with direct 
practical importance, such as aeronautics technology, water 
desalinization, and atmospheric disturbances and weather. The launch of 
Sputnik by the Soviet Union in 1957 led to immediate efforts to expand 
U.S. R&D, science and engineering education, and technology deployment. 
Within months of the Sputnik launch, the National Aeronautics and Space 
Administration (NASA) and the Advanced Research Projects Agency (ARPA) 
in the Department of Defense (DOD) were established. NASA’s core included 
the aeronautics programs of the National Advisory Committee on 
Aeronautics and some of the space activities of DOD; ARPA’s purpose was to 
enable DoD to conduct advanced R&D to meet military needs. 
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Federal appropriations for R&D and for mathematics and science education 
in NSF and other government agencies rose rapidly over the next decade, 
often at double-digit rates in real terms. (See fig. I.1.) During the early 
1950s, the growth in federal funding for health research slowed 
considerably from its fast pace in the immediate postwar years. However, 
in the late 1950s federal support for health research accelerated rapidly. 


From 1966 to 1975, federal support for nondefense R&D dropped nearly 
22 percent in real terms. The costs of the Vietnam War put downward 
pressure on nondefense R&D along with other nondefense discretionary 
spending. The conclusion of NASa’s Apollo program contributed to the 
decline in federal R&D funding during that period, also. Until recently, this 
was the only period in which federal funds for R&D were reduced 
substantially. 
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Source: National Science Foundation, Science Resources Studies Division. 


In the 1970s, new R&D-intensive agencies addressed environmental and 
energy issues. Both the environmental movement and the energy crisis of 
the 1970s, according to some analysts, led to increased public and private 
spending on environmental and energy R&D. The Environmental Protection 
Agency was established in 1970. In 1977, the Energy Research and 
Development Administration and other federal energy-related activities 
were combined to form the Department of Energy, which was given major 
new responsibilities to fund energy-related R&D. 
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Funding 


In the 1980s, the federal role in R&D expanded to enable the United States | 


to compete in a global market. Out of this atmosphere, several programs 
were initiated to provide financial and other incentives for industrial R&D 
and for industrially related R&D conducted at universities or federal 
laboratories. 


Two major factors have influenced federal support for R&D funding in the 
1990s. These factors have been the efforts to reduce the budget deficit and 
the defense drawdown. The Department of Defense and the Department of 
Energy, two of the four largest sources of federal R&D support, have had 
constant-dollar reductions in R&D obligations during the 1990s. In fiscal 
year 1995, Defense accounted for roughly half of all federal R&D 
obligations, down from nearly two-thirds of the total in 1986 at the height 
of the defense build-up that occurred during the Reagan administration. 


At the same time, military-related R&D spending was being curtailed, while 
federal investment in selected civilian R&D activities increased, including 
support for research aimed at improving health and the environment and 
for technology advancement. The Department of Health and Human 
Services, which is a distant second to Defense in terms of total R&D 
support, had the largest absolute increase—$3 billion—in federal R&D 
obligations during the 1990s. The proportion of all U.S. R&D devoted to 
health-related projects has been increasing continuously for nearly a 
decade. The Commerce Department has registered the largest percentage 
increase in federal R&D obligations during the 1990s so far. In addition, the 
federal government, which supplies about three-fifths of all funds used to 
perform R&D on college and university campuses, has been increasing its 
support of academic research continuously since 1982, even after 
adjusting for inflation.!° 


During the early post-World War IJ period, thriving domestic and 
international markets supported large profits and the rapid expansion of 
R&D in both the central laboratories and the divisional laboratories of large 
companies. Central R&D facilities focused increasingly on fundamental 
research in many of these large firms, leaving the development and 
application of new technologies, as well as the improvement of established 
products and processes, to the divisional laboratories. The data on basic 
research for 1953 through 1960 are less reliable than those for later years 
but suggest that the share of total U.S. basic research financed by industry 


13Much of this section is based on Allocating Federal Funds for Science and Technology by the 
Committee on Criteria for Federal Support of Research and Development of the National Academy of 
Sciences, National Academy Press (1995). 
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during the postwar period may well have been at its peak during the 1950s : 


and early 1960s. 


Severe competitive pressures from foreign firms, increases in the real cost 
of capital, and a slowdown in the growth rate of the domestic economy in 
the 1970s have been cited as causes for the apparent decline in the returns 
to R&D investment during the mid-1970s, and the rate of growth in real 
industry expenditures on R&D declined. Industry’s funding of basic 
research shrank, and many of the central research facilities of the giant 
corporations entered a period of budgetary austerity or cutbacks. After a 
resurgence in the early 1980s, the rate of growth in industry-funded R&D 
declined in the late 1980s. 


Industry’s funding of R&D was generally flat between 1991 and 1994; 
several reasons have been cited for the lack of growth in some companies’ 
R&D programs during this time. These include corporate downsizing, 
decentralization (i.e., a shift of R&D activity from corporate laboratories to 
individual business units), and increasing collaboration among industrial 
firms and with partners in academia, in government, in the nonprofit 
sector, and in other countries. The preliminary data for 1995 and 1996 
indicate a slight upswing in industrial R&D funding. 


According to NSF, the most striking recent trend in industrial R&D 
performance has been the increase in the proportion of total R&D funded 
by the companies classified as nonmanufacturing industries. Prior to 1983, 
nonmanufacturing industries accounted for less than 5 percent of the 
industry’s total R&D. That share grew steadily during the next decade so 
that in 1993, nonmanufacturing firms represented more than 25 percent of 
all industrial R&D performed in the United States. Part of this reported 
growth, however, reflects improvements in the NsF’s survey efforts to 
measure R&D in nonmanufacturing industries, in addition to actual growth 
in service sector R&D spending. 


Between 1984 and 1994, some significant changes occurred among the 100 
largest publicly held R&D funding companies, although the four leading 
firms were the same in both 1984 and 1994. During the decade, the number 
of pharmaceutical and computer hardware and software companies 
among the largest R&D funders rose. In contrast, the number of large 
defense contractors and chemical and petroleum companies among the 
largest R&D funders fell. 
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U.S. firms have begun to alter their R&D patterns in response to increasing 
competitive pressures. Firms have shifted a greater portion of their R&D 
resources away from long-term investments and toward shorter-term 
projects. U.S. companies now allocate 22 percent of their R&D spending to 
long-term projects when compared with their Japanese counterparts, who 
devote 50 percent. Increasingly, firms are emphasizing short-term R&D for 
immediate problem-solving or near-term development over basic research; 
and basic research is being directed toward the needs of product 
development and manufacturing teams. Many central research 
laboratories at large companies—such as AT&T, IBM, General Electric, 
Kodak, and Xerox—have been downsized and work more closely with 
product development divisions. They now receive a larger share of their 
operating funds from individual business units rather than general 
corporate funds. 
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